Search for: All records

Training deep learning (DL) models in the cloud has become a norm. With the emergence of serverless computing and its benefits of true pay-as-you-go pricing and scalability, systems researchers have recently started to provide support for serverless-based training. However, the ability to train DL models on serverless platforms is hindered by the resource limitations of today's serverless infrastructure and DL models' explosive requirement for memory and bandwidth. This paper describes FuncPipe, a novel pipelined training framework specifically designed for serverless platforms that enable fast and low-cost training of DL models. FuncPipe is designed with the key insight that model partitioning can be leveraged to bridge both memory and bandwidth gaps between the capacity of serverless functions and the requirement of DL training. Conceptually simple, we have to answer several design questions, including how to partition the model, configure each serverless function, and exploit each function's uplink/downlink bandwidth. In particular, we tailor a micro-batch scheduling policy for the serverless environment, which serves as the basis for the subsequent optimization. Our Mixed-Integer Quadratic Programming formulation automatically and simultaneously configures serverless resources and partitions models to fit within the resource constraints. Lastly, we improve the bandwidth efficiency of storage-based synchronization with a novel pipelined scatter-reduce algorithm. We implement FuncPipe on two popular cloud serverless platforms and show that it achieves 7%-77% cost savings and 1.3X-2.2X speedup compared to state-of-the-art serverless-based frameworks. 

Wide-area soil moisture sensing is a key element for smart irrigation systems. However, existing soil moisture sensing methods usually fail to achieve both satisfactory mobility and high moisture estimation accuracy. In this paper, we present the design and implementation of a novel soil moisture sensing system, named as SoilId, that combines a UAV and a COTS IR-UWB radar for wide-area soil moisture sensing without the need of burying any battery-powered in-ground device. Specifically, we design a series of novel methods to help SoilId extract soil moisture related features from the received radar signals, and automatically detect and discard the data contaminated by the UAV's uncontrollable motion and the multipath interference. Furthermore, we leverage the powerful representation ability of deep neural networks and carefully design a neural network model to accurately map the extracted radar signal features to soil moisture estimations. We have extensively evaluated SoilId against a variety of real-world factors, including the UAV's uncontrollable motion, the multipath interference, soil surface coverages, and many others. Specifically, the experimental results carried out by our UAV-based system validate that SoilId can push the accuracy limits of RF-based soil moisture sensing techniques to a 50% quantile MAE of 0.23%.

 

Abstract Current barriers hindering data-driven discoveries in deep-time Earth (DE) include: substantial volumes of DE data are not digitized; many DE databases do not adhere to FAIR (findable, accessible, interoperable and reusable) principles; we lack a systematic knowledge graph for DE; existing DE databases are geographically heterogeneous; a significant fraction of DE data is not in open-access formats; tailored tools are needed. These challenges motivate the Deep-Time Digital Earth (DDE) program initiated by the International Union of Geological Sciences and developed in cooperation with national geological surveys, professional associations, academic institutions and scientists around the world. DDE’s mission is to build on previous research to develop a systematic DE knowledge graph, a FAIR data infrastructure that links existing databases and makes dark data visible, and tailored tools for DE data, which are universally accessible. DDE aims to harmonize DE data, share global geoscience knowledge and facilitate data-driven discovery in the understanding of Earth's evolution. 

Freshwater stored on land is an extremely vital resource for all the terrestrial life on Earth. But our ability to record the change of land water storage is weak despite its importance. In this study, we attempt to establish a data‐driven model for simulating terrestrial water storage dynamics by relating climate forcings with terrestrial water storage anomalies (TWSAs) from the Gravity Recovery and Climate Experiment (GRACE) satellites. In the case study in Pearl River basin, China, the relationships were learned by using two ensemble learning algorithms, the Random Forest (RF) and eXtreme Gradient Boost (XGB), respectively. The TWSA in the basin was reconstructed back to past decades and compared with the TWSA derived from global land surface models. As a result, the RF and XGB algorithms both perform well and could nicely reproduce the spatial pattern and value range of GRACE observations, outperforming the land surface models. Temporal behaviors of the reconstructed TWSA time series well capture those of both GRACE and land surface models time series. A multiscale GRACE‐based drought index was proposed, and the index matches the Standardized Precipitation‐Evapotranspiration Index time series at different time scales. The case analysis for years of 1963 and 1998 indicates the ability of the reconstructed TWSA for identifying past drought and flood extremes. The importance of different input variables to the TWSA estimation model was quantified, and the precipitation of the prior 2 months is the most important variable for simulating the TWSA of the current month in the model. Results of this study highlight the great potentials for estimating terrestrial water storage dynamics from climate forcing data by using machine learning to achieve comparable results than complex physical models.